A spatialized additive synthesizer

In virtual auditory environments, sound generation is typically based on a two-stage approach: synthesizing a monophonic signal, implicitly equivalent to a point source, and simulating the acoustic space. The directivity, spatial distribution and position of the source can be simulated thanks to signal processing applied to the monophonic sound. A one-stage synthesis/spatialization approach, taking into account both timbre and spatial attributes of the source as low-level parameters, would achieve a better computational efficiency essential for real-time audio synthesis in interactive environments. Such approach involves a careful examination of sound synthesis and spatialization techniques to reveal how they can be connected together. This paper concentrates on the sinusoidal sound model and 3D positional audio rendering methods. We present a real-time algorithm that combines Inverse Fast Fourier Transform (FFT-1) synthesis and directional encoding to generate sounds whose sinusoidal components can be independently positioned in space. In addition to the traditional frequency-amplitude-phase parameter set, partials positions are used to drive the synthesis engine. Audio rendering can be achieved on a multispeaker setup, or in binaural over headphones, depending on the available reproduction system. 1. SINUSOIDAL MODELING In [MAQ86] McAulay and Quatieri present a complete system for speech coding based on sinusoidal analysis/synthesis. The model they describe consists of representing a sound as a sum of time varying sinusoidal components. Amplitude, phase and frequency of partials are extracted from the Short-Time Fourier Transform (STFT) of the original sound, and used as parameters to drive the synthesis engine. The sinusoidal representation is also adapted to a broad range of audio signals, such as musical or environmental sounds. For example sounds produced by vibrating solids are well simulated by sums of damped sinusoids [Cook02]. The partial's frequencies and damping factors can be estimated from the analysis of natural sounds [AKM06] or from analytical solutions of the mechanical model expressed from the object shape and material. Similar techniques are used for modeling harmonic sounds produced by many musical instruments. Sinusoidal analysis/resynthesis allows many effects like filtering, pitch-shifting and time-stretching in computer music. The SDIF Sound Description Interchange Format [SW00] has been created to store, play and manipulate sinusoidal representations. The model has also been significantly extended to take noisy components into account, leading to a 'sinusoid+noise' representation [SS90]. 2. FREQUENCY-DOMAIN SINUSOIDAL SYNTHESIS Sinewaves synthesis can be done either in time or frequency domain. In [RD92] Depalle and Rodet propose a complete frequency-domain additive synthesizer. Compared to time-domain sinewave synthesis, their method allows to drastically reduce computational complexity, without affecting perceived sound quality. This algorithm is the kernel of the 3D synthesis engine presented in part 4. It consists of creating Short Time Spectra (STS) by successively adding a spectral motif, with specific amplitude and phase, at desired frequency for each partial. STS are inverse fast Fourier transformed (IFFT) and overlap-added (OLA) to obtain the synthetic sound. Let U(f) be the STS and W(f) the spectral motif. Amplitude, frequency and phase parameters are noted [Am,Fm,Pm]. U(f) is given by: ∑ = − = M m m m j m F f W P e A f U 1 ) ( ) ( where M is the total number of partials. The sampled version U[k] of this spectrum is calculated and IFFT-OLA. This procedure is illustrated in figure 1, for M disjoint partials. Figure 1: frequency-domain sinewaves synthesis [RD92] The spectral motif is the complex spectrum of a time-domain window w(t). The crucial reduction factor on the algorithm's computational complexity relies on the choice of this window. If window's energy is sufficiently concentrated in a narrow low Proceedings of ICoMCS December 2007 Page 167 The inaugural International Conference on Music Communication Science 5-7 December 2007, Sydney, Australia http://marcs.uws.edu.au/links/ICoMusic frequency band, then the complex spectrum can be truncated without losing much information. It results in reducing the number of complex multiplications and additions necessary to add each partial, because adding the narrow-band spectral motif involves modifications of the sampled STS U[k] only for a few frequency bins. However, to obtain the perfect reconstruction of a constant amplitude sinewave, w[n] (sampled version of w(t)) should also satisfy the condition: where L is the synthesis stride. This constraint is compatible with a narrow-band spectral motif only when the stride is small compared to the window size. A 75% overlap can be used but it does not optimize the algorithm efficiency. Rodet and Depalle propose to use a second window after the IFFT that allows to reduce the overlap to 50% [RD92]. The drawback, important for our application, is the difficulty to filter the STS in the frequency domain before the IFFT. Consequently we choose to keep the simple approach of a single window with a small synthesis stride. We use a "digital prolate spheroidal" window [VBM96] weighted to satisfy the perfect reconstruction constraint with 75% overlap between blocks. The resulting spectral motif is sharp so that synthesizing sinusoidal blocks (of any size) only requires 7 points per partial when building the STS. 3. 3D SOUND SOURCE POSITIONING For versatility, a 3D sound synthesizer should be independent of the available reproduction setup and compatible with most of existing positional audio methods and their directional encoding/decoding scheme. This part quickly reviews techniques to position a monophonic point source in a virtual 3D space. A more detailed overview of 3D audio encoding and rendering can be found for example in [JLP99]. The source, placed in the spherical coordinate system shown in figure 2, is assumed to radiate a plane wave in the listener direction. Figure 2: egocentric spherical coordinate system Among 3D positional audio methods, some aim at simulating the localization cues at the ear canal entrance (binaural techniques), other at reproducing the sound field properties at the "sweet spot" (Discrete panning techniques [Sch71] [Pul97], Ambisonics [MM95]) or in a relatively extended area (Higher Order Ambisonics, Wave Field Synthesis [DNM03]). Binaural techniques are mainly dedicated to headphone reproduction but can be extended to multispeaker configurations by means of crosstalk cancellation. Other methods require multichannel playback systems. Apart from Wave Field Synthesis, which uses time differences to create virtual sources, all positioning methods can be implemented by applying a vector of gain factors to the original monophonic sound. This is the key point to connect directional encoding to the FFT-1 synthesis algorithm described in part 2. n n iL w i ∀ = + ∑ +∞